Beyond Convex Optimization: Star-Convex Functions

We introduce a polynomial time algorithm for optimizing the class of star-convex functions, under no Lipschitz or other smoothness assumptions whatsoever, and no restrictions except exponential boundedness on a region about the origin, and Lebesgue measurability. The algorithm’s performance is polynomial in the requested number of digits of accuracy and the dimension of the search domain. This contrasts with the previous best known algorithm of Nesterov and Polyak which has exponential dependence on the number of digits of accuracy, but only n dependence on the dimension n (where ω is the matrix multiplication exponent), and which further requires Lipschitz second differentiability of the function [13]. Star-convex functions constitute a rich class of functions generalizing convex functions, including, for example: for any convex (or star-convex) functions f , g, with global minima f(0) = g(0) = 0, their power mean h(x) = ( f(x)+g(x) 2 )1/p is star-convex, for any real p, defining powers via limits as appropriate. Star-convex functions arise as loss functions in non-convex machine learning contexts, including, for data points Xi, parameter vector θ, and any real exponent p, the loss function hθ,X(θ̂) = (∑ i |(θ̂ − θ) ·Xi| )1/p , significantly generalizing the well-studied convex case where p ≥ 1. Further, for any function g > 0 on the surface of the unit sphere (including discontinuous, or pathological functions that have different behavior at rational vs. irrational angles), the star-convex function h(x) = ||x||2 · g ( x ||x||2 ) extends the arbitrary behavior of g to the whole space. Despite a long history of successful gradient-based optimization algorithms, star-convex optimization is a uniquely challenging regime because 1) gradients and/or subgradients often do not exist; and 2) even in cases when gradients exist, there are star-convex functions for which gradients provably provide no information about the location of the global optimum. We thus bypass the usual approach of relying on gradient oracles and introduce a new randomized cutting plane algorithm that relies only on function evaluations. Our algorithm essentially looks for structure at all scales, since, unlike with convex functions, star-convex functions do not necessarily display simpler behavior on smaller length scales. Thus, while our cutting plane algorithm refines a feasible region of exponentially decreasing volume by iteratively removing “cuts”, unlike for the standard convex case, the structure to efficiently discover such cuts may not be found within the feasible region: our novel star-convex cutting plane approach discovers cuts by sampling the function exponentially far outside the feasible region. We emphasize that the class of star-convex functions we consider is as unrestricted as possible: the class of Lebesgue measurable star-convex functions has theoretical appeal, introducing to the domain of polynomial-time algorithms a huge class with many interesting pathologies. We view our results as a step forward in understanding the scope of optimization techniques beyond the garden of convex optimization and local gradient-based methods. ar X iv :1 51 1. 04 46 6v 2 [ cs .D S] 1 1 M ay 2 01 6